Cycle Policy Level-03 Activity Execution Stats

Cycle Test Criteria

12.2 - Cycle performance test was conducted for Policy level processing. Cycle benchmark tests were executed to identify and verify the significant improvements or degradation's in this current release.

The performance plan includes varied configuration for Policy level transactions. Cycle tests were conducted on 5,300,000 policy data-set with ~52,000,000 pending activities to process. The scenarios picked for cycle performance testing includes Term and VDA PIT policies consisting of various activities matching the real time activity processing. The chosen scenarios in creating master policy data-set were close to production like policy contracts and plans.

Server Properties

Cycle.Properties - Cycle Client

Batch Size:
- Maintain batch size larger in the range of 5,000-10,000 or even more, if the heap is not being compromised. Remember heap should be at least 30% free.
- cycle.batchSize is the maximum number of Cycle tasks that will process in the Cycle Grid. Consider the following:
  - Min must be
    - Batch Size = sum of threads allocated to cycle agents (example: If we have 4 Cycle agents with 50 threads each, then Batch size will be 200).
    - To ensure that the threads are always running, set the Batch Size to twice the number of threads in the Cycle Grid and then multiply it with the group size.
- The limitation on the number of tasks running in the cluster is dependent on the available memory in the cluster, not the thread pool sizes. The number of threads in the grid is only a theoretical estimate of how many tasks will be running at one time in the grid. Increasing the batchSize has the added benefit of reducing locking in the database when retrieving new tasks, which will increase performance. Setting the batchSize to 1000, 10000 or more is possible, depending on the memory in the cluster.
Cache Processing threads: Coherence Concurrent provides a facility to dispatch tasks, either a Runnable or a Callable, to a Coherence cluster for execution. Executors that actually execute the submitted tasks are configured on each cluster member by defining one or more named executors within a cache configuration resource. We have set it to 50 threads.
Cycle Group Size:
- It determines the number of Cycle work items (Policies or Clients) to group together and execute on a single thread in the cluster.
- The groupSize has an impact on the number of tasks that are submitted to the grid for processing. If the batchSize is set to 10000, and the groupSize is set to 10, then 1000 actual tasks will be submitted to the grid for processing, each task containing 10 work items to be completed. Consider the groupSize value when determining the batchSize.
- Increase this value in order to group together tasks and speed up task submission and processing. The grid will process 10000 work items grouped in sets of 10 faster than 10000 individual work items.
- Keep Group Size lower, usually under 30. The group size will group activities for a single thread submission.
Cycle Period:
- cycle.period is the number of seconds that the Cycle Agent will wait before checking for additional work.
- Set it higher in order to avoid pinging the database too frequently for work. Frequent trips to the database, especially across multiple Cycle Agents, can impact performance.
- Set it low enough that you do not starve the Cycle Agent. If the threads in the Cycle Agent run out of work, they will not do anything until cycle.period expires and work is checked again.
- Keep this property lower, usually set to 5 seconds. This will let Cycle sleep for 5 seconds and retry replenishing grid every 5 seconds

Cycle.Properties - Cycle Web

Application Mode: PRODUCTION
Application Resource Cache Timeout: -1

Cycle Hardware Specs

Cycle on Oracle DB

Specs

Oracle DB Parameter Configuration
sessions	5824
processes	3840
memory_target	0
memory_max_target	0
session_cached_cursors	2000
cursor_sharing	FORCE
commit_logging	IMMEDIATE
commit_wait	NOWAIT
open_cursors	1000
sga_max_size	50G
sga_min_size	0
sga_target	50G

Note:
1. Added redo log files with 3 groups of 3 redo files of 8GB each.
2. Reorganized tables with large amounts of data.
3. Gathered schema level statistics before each cycle execution.

Application Server Specs
No. of OCPUs	16
No. of Computes (VMs)	2
No. Of Cycle Agents (JVMs)	8 ( 4 in each VM)

JVM Specs
Heap size	6 GB
GC	G1GC

Cycle Specs
cycle.period	5
cycle.batchSize	24000
cycle.groupSize	30
Coherence Specs
cycle.thread.count ( in coherence cache file )	50
total thread count	400 → (50 * 8 as there are 8 cycle nodes)
executor	fixed
name	Single-Cache

Data Specs - Before Running Cycle
Policy Volume	5.3M
Activity Volume	52M

DB connection pool size

#	Data Source Name	Parameters
1	ADMINSERVERDS ADMINSERVERSEARCHDS ADMINSERVERRESOURCEDS ADMINSERVERREADONLYDS	maxActive = 300 maxIdle = 50 minIdle = 10 initialSize = 20 maxWait = 10000

Data Source Name

Parameters

ADMINSERVERDS

ADMINSERVERSEARCHDS

ADMINSERVERRESOURCEDS

ADMINSERVERREADONLYDS

maxActive = 300

maxIdle = 50

minIdle = 10

initialSize = 20

maxWait = 10000

Business Day Processing
Transaction Type	Plan Name	Count
BankDraft	Guaranteed Level Premium Term	2541
BankDraftGetSuspense	Guaranteed Level Premium Term	2541
BankDraftStart	Guaranteed Level Premium Term	1263173
BankDraftStop	Guaranteed Level Premium Term	2651
COIPayment	Guaranteed Level Premium Term	2651
CoverageCalculation	Guaranteed Level Premium Term	3430
Disbursement	Guaranteed Level Premium Term	2651
Issue	Guaranteed Level Premium Term	2563
Premium	Guaranteed Level Premium Term	5243
WaiverActivate	Guaranteed Level Premium Term	2651
WaiverDeactivate	Guaranteed Level Premium Term	1260430
AutoPayStart	Variable Deferred Annuity-PIT	5138
BeneficiaryConfirmationRequest	Variable Deferred Annuity-PIT	3121
Commission	Variable Deferred Annuity-PIT	7663
DeathBenefitPayout	Variable Deferred Annuity-PIT	3118
DeathNotify	Variable Deferred Annuity-PIT	3129
Disbursement	Variable Deferred Annuity-PIT	3118
InitialPremium	Variable Deferred Annuity-PIT	7663
Issue	Variable Deferred Annuity-PIT	11928
PolicyDeathNotify	Variable Deferred Annuity-PIT	3373
PolicyProofOfDeath	Variable Deferred Annuity-PIT	3263
Submit	Variable Deferred Annuity-PIT	7715
SystematicWithdrawalStart	Variable Deferred Annuity-PIT	5138
AnniversaryProcessing	Variable Deferred Annuity-PIT T-Y	3

Performance Results

Note: Above mentioned results are purely based on our internal test scenarios, our internal transaction configurations, SQL and hardware tuning. The results may vary depending on customer's own configuration, data volumes, hardware specifications and db and application server tuning.